Skip to content

Migrated MariaDB, MySQL, PostgreSQL tests to use testcontainers to simplify local setup - #2011

Merged
koperagen merged 12 commits into
masterfrom
testcontainers
Aug 17, 2026
Merged

Migrated MariaDB, MySQL, PostgreSQL tests to use testcontainers to simplify local setup #2011
koperagen merged 12 commits into
masterfrom
testcontainers

Conversation

@koperagen

@koperagen koperagen commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Tests still will not be run on CI for now. But it'll be now possible to run MariaDB, MySQL, PostgreSQL in one click, providing you have docker installed on PC. It's the only prerequisite.
Let's start with those and migrate imbd, mssql example after

For reference. Looks like their JUnit4 examples are a bit outdated and nowadays modern "container" classes do not implement "testrule" class, so we use manual lifecycle management instead
https://java.testcontainers.org/modules/databases/mysql/
https://java.testcontainers.org/modules/databases/mariadb/
https://java.testcontainers.org/modules/databases/postgres/

@koperagen
koperagen requested a review from zaleslaw July 28, 2026 15:54
st.setTime(16, java.sql.Time(System.currentTimeMillis()))
st.setTimestamp(14, SqlTimestamp(System.currentTimeMillis()))
st.setTimestamp(15, SqlTimestamp(System.currentTimeMillis()))
st.setTime(16, SqlTime(System.currentTimeMillis()))

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automatic refactorings break such fully qualified links :( I recovered it as import alias, should survive file moves etc

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

It currently introduces a dependency-resolution issue in the version catalog and a Kotlin compilation error from an unused import.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

This PR migrates the JDBC integration tests for MariaDB/MySQL/PostgreSQL to Testcontainers so contributors can run them locally with Docker, while keeping CI’s default test task free of Docker-dependent tests.

Changes:

  • Added Testcontainers dependencies to the version catalog and dataframe-jdbc test configuration.
  • Updated MariaDB/MySQL/PostgreSQL test suites to start/stop containers in @BeforeClass/@AfterClass rather than relying on localhost DBs.
  • Added a dedicated testcontainersTest Gradle task and excluded the Testcontainers package from the default test task.
File summaries
File Description
gradle/libs.versions.toml Adds Testcontainers version + catalog entries for DB modules.
dataframe-jdbc/build.gradle.kts Adds Testcontainers test deps; excludes Docker tests from test and introduces testcontainersTest.
dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/testcontainers/postgresTest.kt Switches Postgres tests to a managed PostgreSQLContainer.
dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/testcontainers/postgresConnectionUrlTest.kt Reworks Postgres URL parsing tests to run against a container.
dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/testcontainers/mysqlTest.kt Switches MySQL tests to a managed MySQLContainer.
dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/testcontainers/mariadbTest.kt Switches MariaDB tests to a managed MariaDBContainer.
dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/testcontainers/images.kt Centralizes Docker image tags used by the Testcontainers tests.
Review details

Comments suppressed due to low confidence (2)

dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/testcontainers/mysqlTest.kt:20

  • Unused import org.junit.Ignore will fail Kotlin compilation (unused imports are errors). The @Ignore annotation was removed, so this import should be removed too.
    dataframe-jdbc/src/test/kotlin/org/jetbrains/kotlinx/dataframe/io/testcontainers/postgresTest.kt:30
  • Unused import org.junit.Ignore will fail Kotlin compilation (unused imports are errors). The @Ignore annotation was removed, so this import should be removed too.
  • Files reviewed: 7/7 changed files
  • Comments generated: 1
  • Review effort level: Low

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Comment thread gradle/libs.versions.toml
Comment on lines +125 to +127
testcontainers-postgresql = { group = "org.testcontainers", name = "testcontainers-postgresql", version.ref = "testcontainers" }
testcontainers-mysql = { group = "org.testcontainers", name = "testcontainers-mysql", version.ref = "testcontainers" }
testcontainers-mariadb = { group = "org.testcontainers", name = "testcontainers-mariadb", version.ref = "testcontainers" }
@zaleslaw

Copy link
Copy Markdown
Collaborator

@copilot will this work on Windows or on Team City?
Could we refactor it and keep both the local and test container versions of tests, but call them in two forms - Docker Container and run on Local Database, don't mind about @ignore tag - it doesn't play any role

@zaleslaw

Copy link
Copy Markdown
Collaborator

https://gh.io/copilot-coding-agent-docs will this work on Windows or on Team City?

@Jolanrensen

Jolanrensen commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Did you test it with it already on TC?
You can configure the gradle build-step to run in a certain Docker container like dockerImage = "cimg/android:2026.02.1" I assume, like this we could make test-containers work as well.

See https://www.jetbrains.com/help/teamcity/2026.1/gradle.html?TeamCity%20Documentation#Container+Settings and https://www.jetbrains.com/help/teamcity/2026.1/container-wrapper.html?TeamCity%20Documentation

(locally, they seem to run well, btw!)

@koperagen

koperagen commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator Author

@copilot will this work on Windows or on Team City? Could we refactor it and keep both the local and test container versions of tests, but call them in two forms - Docker Container and run on Local Database, don't mind about @ignore tag - it doesn't play any role

bad copilot : ( why did it have to commit things to my branch and break CI. Besised, i'd do abstract test class and have LocalMariaDb tests + DockerMariaDb tests instead

@koperagen

Copy link
Copy Markdown
Collaborator Author

Did you test it with it already on TC?
will this work on Windows or on Team City?

On TeamCity eventually yes, need to configure teamcity to run on agents that have docker daemon

@zaleslaw

Copy link
Copy Markdown
Collaborator

Great job! Will test it next week on both MacOS and WIndows, locally and not, and it will be shipped!

@zaleslaw

zaleslaw commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Is it possible to migrate MSSQL ? Also, it's very painful to test

@koperagen

koperagen commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator Author

Is it possible to migrate MSSQL ? Also, it's very painful to test

i believe so, but i want to make sure it's ok for us because we need to accept some license agreement to use mssql

if overall we're ok with migrating to testcontainers i'll proceed with mssql

Due to licencing restrictions you are required to accept an EULA for this container image. To indicate that you accept the MS SQL Server image EULA, call the acceptLicense() method, or place a file at the root of the classpath named container-license-acceptance.txt, e.g. at src/test/resources/container-license-acceptance.txt. This file should contain the line: mcr.microsoft.com/mssql/server:2017-CU12 (or, if you are overriding the docker image name/tag, update accordingly).

Please see the microsoft-mssql-server image documentation for a link to the EULA document.

Comment thread dataframe-jdbc/build.gradle.kts

@zaleslaw zaleslaw left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Include in the Readme.md information and links about requirement to set up the Docker env for test conteainers with minimum technical requirements for 3 OS or link to that

  • describe the idea of test separation (it's complicated)

`## :dataframe-jdbc

This module, published as dataframe-jdbc, contains all logic and tests for DataFrame to be able to work with
JDBC data sources.

See Read from SQL databases for more information
about how to use it.

Testing

The JDBC tests are split into three tiers by how they obtain a database. This keeps the default build
fast and dependency-free, while still allowing full coverage against real database engines.

1. H2 tests (default suite)

These run against an in-memory H2 database using its compatibility modes
(MODE=PostgreSQL, MODE=MySQL, MODE=MariaDB, MODE=MSSQLServer, etc.). They need no Docker and no external
server, so they are part of the regular test task and run in CI on every build.

Location: src/test/kotlin/.../io/h2/ (e.g. PostgresH2Test, MysqlH2Test, MariadbH2Test, MssqlH2Test).
Together with the SQLite/DuckDB tests they form the baseline coverage that always runs.

./gradlew :dataframe-jdbc:test

2. Testcontainers tests (real DB engines via Docker)

These spin up real PostgreSQL, MySQL, and MariaDB engines in Docker using
Testcontainers. They give the highest-fidelity coverage but require a running
Docker daemon, so they are excluded from the default test task and run under a dedicated Gradle task.

Location: src/test/kotlin/.../io/testcontainers/ (package org.jetbrains.kotlinx.dataframe.io.testcontainers).
The container image tags are configured in build.gradle.kts via BuildConfig
(POSTGRES_IMAGE, MYSQL_IMAGE, MARIADB_IMAGE).

Requirements: a running Docker daemon.

./gradlew :dataframe-jdbc:testcontainersTest

3. Local DB tests (servers on localhost)

These connect to database servers you run yourself on localhost (e.g. Postgres on 5432, MySQL, MariaDB, MSSQL).
They are meant for local, manual verification against a specific server and are excluded from the default
test task
.

Location: src/test/kotlin/.../io/local/. Only classes whose names end with LocalTest
(PostgresLocalTest, MySqlLocalTest, MariadbLocalTest, PostgresConnectionUrlLocalTest) are picked up by the
dedicated task. The connection URLs/credentials are hardcoded in each test class, so start a matching server
before running them.

./gradlew :dataframe-jdbc:localDbTest

How the exclusions are wired

The default test task explicitly excludes both the Testcontainers and local-DB tests, and each is exposed as its
own task (see build.gradle.kts):

private val testcontainersTests = "org.jetbrains.kotlinx.dataframe.io.testcontainers.*"
private val localDbTests = "org.jetbrains.kotlinx.dataframe.io.local.*LocalTest"

tasks.test {
    filter {
        excludeTestsMatching(testcontainersTests)
        excludeTestsMatching(localDbTests)
    }
}

tasks.register<Test>("testcontainersTest") { /* includeTestsMatching(testcontainersTests) */ }
tasks.register<Test>("localDbTest")        { /* includeTestsMatching(localDbTests) */ }

So: H2 (and SQLite/DuckDB) run by default; Testcontainers and local-DB tests are opt-in via their own tasks.`

@zaleslaw

zaleslaw commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Yes, we can proceed with MSSQL Testcontainers.

  1. MSSQL is used only during tests and is not distributed with Kotlin DataFrame.
  2. The SQL Server Docker image remains a separate external component, so it does not affect our Apache 2.0 license.
  3. acceptLicense() only sets ACCEPT_EULA=Y inside the container; it does not send TeamCity environment data to Microsoft.

Our purpose is clear: use MSSQL only for automated integration testing.

@koperagen

@zaleslaw

zaleslaw commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Could help with, but need to verify #563

@koperagen koperagen self-assigned this Aug 13, 2026
@koperagen koperagen linked an issue Aug 13, 2026 that may be closed by this pull request
@koperagen
koperagen merged commit da16c44 into master Aug 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Setup ability to run JDBC tests in testcontainers env

4 participants